Virtual suction tool

ABSTRACT

A touch-based computer graphics user interface enhances a stylus pen with virtual suction to create a virtual vacuum or suction tool. The virtual tool may simulate a physical world suction tool and/or provide capabilities not available in the real world. The virtual suction tool allows the user to feel as if he or she has transcended the boundary between the virtual world and the real world by physically holding in his or her hand an object that is able to interact with and virtually attract and contain items removed from or expellable into the virtual world. The resulting highly intuitive user interface provides fun and exciting video game play and has other applications as well.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a nonprovisional application claiming the benefit ofpriority from application Ser. No. 60/746,390 filed on May 4, 2006 andentitled “Virtual Suction Tool”, incorporated herein by reference as ifexpressly set forth. This application is related to commonly assignedcopending application Ser. No. 11/243,183 filed: Oct. 5, 2005, entitled“Driving Game Steering Wheel Simulation Method and Apparatus”.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

FIELD

The technology herein relates to user interfaces for interacting with acomputer and more specifically to graphical 3D user interfaces. Stillmore specifically, the technology herein relates to 3D computer displaysand/or computer generated sound effects that enhance the apparentcharacteristics of a handheld object to provide virtual toolcapabilities such as for example, suction, reverse suction and/or objectcontainment.

BACKGROUND AND SUMMARY

Tools help us accomplish our daily tasks. A parent uses an electric orgas range, pots and pans, and spoons, spatulas and other implements tocook a meal for his or her family. A carpenter uses planes, saws,hammers and levels to cut pieces of wood to size and fasten them in. Amechanic uses wrenches, gear pullers, jacks, lifts and other tools todisassemble and reassembly a car engine. A gardener uses rakes, hoes,clippers and other implements to help nurture plants.

Just as in the real world, tools are also important in virtual realty,augmented reality and computer graphic simulations. Therefore, much workhas been done in the past to develop efficient ways for a human user tocontrol virtual tools displayed on a screen. In some cases, nearly thesame controls a human user will operate in the real world can be used tocontrol a computer graphics simulation of the same tool.

For example, an aircraft simulator used to train pilots often resemblesnearly exactly a cockpit of an airplane. Large display screens are usedto display what a pilot would see outside the window as if her or shewere flying an actual airplane. The pilot in training operates the samecontrols found in a real cockpit to control the virtual aircraftsimulation.

As another example, surgeons are sometimes trained using computersimulators that simulate the way the human body responds to surgicalprocedures. In such surgical simulations, a virtual scalpel may be usedto cut tissue, a virtual suction device may be used to suction awayblood and debris, and a virtual suture may be used to sew up thesimulated wound. Such simulations allow surgeons to try new surgicaltechniques—oftentimes using real world controls that very closelyresemble the implements they will use in the operating room—withoutrisking the health and well being of actual human patients. Althoughthere is no substitute for actual real world experience, suchsimulations can be useful as a first step toward training new surgeons.

Although designers of expensive aircraft, surgical and other simulatorshave the luxury of advanced, authentic controls and other input devices,designers of video game systems and other consumer based computersystems are often presented with a different set of challenges. Forexample, the conventional home video game platform, personal computer orhandheld video game playing platform often has a relatively limited setof standard, general-purpose input devices the human player can use tomanipulate video game action. Similarly, the computer keyboard and mouseof a conventional personal computer are generally used to control nearlyall game functions of video games played on personal computers. SeriousPC gamers will sometimes invest in a joystick peripheral device to allowmore advanced directional control of certain games. However, suchadditional peripheral devices become impractical when gamers attempt toplay games on portable platforms such as portable video gaming systems,cellular telephones with graphics capabilities, personal digitalassistants, pocket PCs, and a host of other devices. In those contexts,the control interface that is available with the device out of the boxis typically the one most players end up using for nearly all of theirinteractions. A challenge is to make maximum use of such input devicesto create user-friendly, effective graphical user interfaces.

But while designers of video games, virtual and augmented reality basedsystems, and other virtual computer-generated environments may havecertain limitations, they also have an additional degree of freedom.Just like in the real world, a character within a video game or othercomputer-generated environment may also use tools to accomplish certainvirtual goals. But because computer generated environments are virtual,developers are not limited to real world tools. They can create new orfanciful tools not possible in the real world.

For example, the main character Link in Nintendo's highly successfulLegend of Zelda video game series uses a sword, a musical instrumentcalled an ocarina, a “windwaker” stick, a bow and arrow, a boomerang, atelescope, a fishing rod, and various other tools to accomplish certainresults within the games. Some of these tools resemble real worldobjects, but within the game they often have magical properties. Forexample, Link can travel through time and otherwise magically manipulatehis environment by playing certain songs on his ocarina.

Another interesting example is the “Poltergust 4000” vacuum cleanerstrapped to the game character Luigi's back in Nintendo's renowned“Luigi's Mansion” video game. While this virtual vacuum cleaner has someresemblance to the kind of vacuum cleaner you clean your carpets with,it has amazing properties no real world vacuum cleaner would everexhibit.

The human game player can operate Luigi's “Poltergust 4000” vacuumcleaner just like a real vacuum cleaner to “suck” (suction mode) or“blow” (exhaust mode). In the suction mode, the virtual vacuum cleanersucks up objects within Luigi's virtual environment including but notlimited to ghosts, fire spirits, water, ice cubes and balls. In theexhaust mode, the vacuum cleaner can exhaust some of these vacuumed-upitems in fanciful ways. For example, if a fire spirit was the last itemLuigi's vacuum cleaner sucked up, then operating Luigi's vacuum cleanerin the exhaust mode transforms the vacuum cleaner into a flame thrower.Similarly, if the last item the game player caused Luigi's vacuum playerto suck up was water from a fountain, then Luigi's vacuum cleaner willin the exhaust mode act as a fire hose that emits a virtual water spray.This creative way of enhancing the functionality of an everydayappliance with magical or extraordinary properties has found great favoramong millions of video game players throughout the world.

Despite the limitations in low cost video game and other consumersystems, there is an interesting relationship between the input devicesthe human game player uses to control a video game or other virtualenvironment, and the tools within the video game or other virtualenvironment used to accomplish objectives. In video games such as theLegend of Zelda and Luigi's Mansion, the human player uses buttonsand/or joy sticks on a handheld controller (e.g., the “C stick”) tocontrol the operation of the tools the game characters use within thegame. For example, in the Luigi's Mansion game, the human player cancontrol whether Luigi's virtual “Poltergust 4000” vacuum cleaner is inthe suction mode, the exhaust mode, a flashlight mode, or no mode bymanipulating buttons and/or joysticks on the Nintendo GameCube handheldcontroller. The handheld controller functions as a control panel forLuigi's virtual vacuum cleaner, allowing the human video game player tocontrol where the “Poltergeist 4000” vacuum cleaner is aiming and whatmode it is operating in. The same joysticks and buttons would be used tocontrol direction, acceleration, turning radius and other attributes ofa vehicle in a driving or flight simulator game.

Video game and computer graphics designers are constantly searching fornew and innovative ways to make interaction with virtual, simulated andaugmented reality worlds more efficient, cost-effective and enjoyable.Recently, touch screen based user interfaces have been introduce in thecontext of video game systems, banking terminals and other computerdevices as a way to interact more efficiently with a graphical-basedcomputer system. However, further improvements are possible anddesirable.

The technology herein uses a 3D graphical computer interface to enhancethe function of a physical object used to interact with a video game,computer simulation or any other graphical presentation. The object ahuman user holds in his or her hands may be relatively static ornon-electronic, and yet may—through interaction with a specialized userinterface provided via a multimedia (e.g., audio and/or graphical video)display presentation—appear to possess enhanced functionalcharacteristics and properties. An ordinary handheld object may, throughinteraction with a computer graphical interface, appear, for example, totake on extraordinary or even magical virtual tool characteristics.

In one specific exemplary illustrative non-limiting implementation, thestylus used to interact with a touch screen based video game interfacemay, through simulation of enhanced functions on a computer graphicsscreen, appear to take on characteristics of a virtual vacuum or suctiontool. In this exemplary illustrative non-limiting implementation, thestylus itself may be a conventional elongated stick-like implement suchas a piece of inert hard plastic. Such a stylus in one exemplaryillustrative non-limiting implementation has no internal cavity, noelectronics, and substantially no mechanical function other than toindicate touching positions on a touch pad or touch screen. An exemplaryillustrative non-limiting implementation of the technology herein usescomputer graphics and computer-generated sound to animate interactionwith such a conventional stylus to provide suction and exhaustproperties of a virtual suction tool.

For example, unlike a real suction tool which may have relativelylimited ability to pick up and put down real world objects in terms ofdimensions, object characteristics and the like, the virtual suctiontool provided by one exemplary illustrative non-limiting implementationof the technology herein can have fanciful object manipulationcapabilities such as infinite capacity to suck up objects of all sorts.In a puzzle game for example, the virtual suction tool may be able tosuck up any number of bubbles, blocks or other objects within the game.As the human user moves the tip of the virtual suction tool over theobjects, each object may disappear in a way that creates an impressionthat the stylus is sucking the object up into an internal cavity orreservoir within the stylus. For example, as the object is beingvirtually suctioned, portions of the object closest to the tool maydeform on the screen to make it appear as if the object has beenpartially sucked into the virtual vacuum tool. Once the stylus virtuallycontains the entire object that has been sucked up, the object maydisappear from the display.

In a further exemplary non-limiting illustrative implementation, anoise, such as a sucking or whooshing noise, may accompany each captureof an object, so that the player feels as if the stylus has actuallysucked the object from the screen. Similarly, an exhaust sound canaccompany virtual “exhaust” of an object back into the virtualenvironment. The human player hearing a computer-generated suction soundcreates the impression that the virtual suction tool stylus or otherobject has sucked up the object and now “contains” the object. Thestylus or other object may virtually “store” any number of digitalobjects, the capacity being limited only by the game design.

In example illustrative non-limiting implementations, the human playermay move the stylus to a different position in the game and expel thesame (or transformed) object(s) back into the game environment.Accompanying exhaust sounds and graphical effects that make it appear asif the stylus is actually expelling them into the game environment.Alternatively or in combination, the object may be transformed by thevirtual suction device (e.g., a red block may be transformed into fire)in the process of being virtually exhausted or expelled into the gameenvironment.

The technology herein is thus capable of transforming an everyday commonphysical object such as an inert stylus into a device that may but doesnot necessarily need to exist in the real world. Exemplary illustrativenon-limiting implementations thus provide methods and apparatus for aplayer to cross the boundary between the virtual world and the realworld. Through displayed virtual world images and sounds, the playergets the impression that a virtual object can be brought into andmanipulated in the real world. He similarly has the feeling that thephysical object he is holding in his hand has virtual tool properties,functions and/or characteristics defining the manner in which itinteracts with the virtual world. For example, in the case of a stylusbeing used as a virtual suction tool, the player may see the objectdistort and shrink as it is “sucked into” the stylus—even though thestylus is incapable of applying suction in the real world, is notconnected to a vacuum pump or vacuum reservoir, and has no internalreservoir or other capacity for containing objects or material.

Accompanied by a suction-like sound, the player is given the impressionthat the plastic stylus has actually pulled and captured an object fromthe virtual world, bringing it into his world. If the player decides toempty the contents of the virtual suction tool back into the virtualworld, an animation of the object being expelled from the stylusaccompanied by a distinctive “exhaust” sound gives the player theimpression that the object (or a transformed version of the object) hasbeen returned from the real world back into the virtual world.

Particularly advantageous exemplary illustrative non-limitingimplementations make use of a touch screen to provide the boundarybetween the real world and the 3D virtual world. A stylus, finger orother object interacting with a touch screen provides a convenient wayto indicate position in the virtual world where the interaction is tooccur. The fact that the stylus, finger or other object is directly incontact with the touch screen allows virtual world graphics at the pointof contact to appear to be interacting directly with the physicalobject. Such an “Alice Through the Looking Glass” property of thevirtual suction tool is also highly intuitive and natural and easy touse.

Other alternative arrangements using other pointing or positionindicating arrangements, with or without distance between the object andthe display, can also be provided. Thus, the physical object in oneexemplary illustrative non-limiting implementation can be a touch screenstylus, but the physical object can be any object capable of interactingwith a 3D or other graphics or other display system including but notlimited to remote control devices, specially designed objects that havephysical appearance of tools or weapons, etc. Since the physical objectused for interaction is given virtual or simulated characteristics,capabilities and/or functions, the resulting virtual tool can differfrom and/or exceed functions, capabilities and/or characteristics ofactual physical objects existing in the real world. For example, avirtual suction tool can have properties that differ from and/or exceedthe properties of a real world suction tool. An inexpensive, inert orother piece of plastic or other material can serve as a variety ofdifferent tools for a variety of games, each tool resembling, simulatingor exceeding the bounds of any real world equivalent. The types andfunctions of each virtual tool are limited only by the imagination ofthe game developer. Because the game player has direct control over thevirtual objects and can manipulate them seemingly in the real world, theplayer gains a high degree of control and a very interesting andengaging gaming experience. In a gaming context, the player feels as ifhe is directly manipulating the game object and so has a closeconnection to the game experience.

BRIEF DESCRIPTION OF THE DRAWINGS

These and other features and advantages will be better and morecompletely understood by referring to the following detailed descriptionof exemplary non-limiting illustrative embodiments in conjunction withthe drawings of which:

FIGS. 1A-1C are exemplary representations of an exemplary illustrativeimplementation of a virtual suction tool;

FIG. 2A is an exemplary external view of an exemplary illustrativenon-limiting game apparatus for executing a game program providingvirtual suction tool object control;

FIG. 2B is a block diagram showing an internal structure of theexemplary game apparatus;

FIG. 2C is a flowchart of an exemplary illustrative non-limitingimplementation of software-controlled virtual tool functionality;

FIG. 3A is an exemplary representation of a movable game character in avirtual gamespace encountering an obstacle;

FIG. 3B is an exemplary representation of a virtual suction toolinteracting with digital objects in a virtual gamespace to removeobjects from the virtual gamespace;

FIG. 3C is an exemplary representation of the virtual suction tooldepositing previously removed digital objects into a virtual gamespaceto aid a character in overcoming an obstacle;

FIG. 4A is another exemplary representation of a movable game characterin a virtual gamespace encountering an obstacle;

FIG. 4B is another exemplary representation of a virtual suction toolinteracting with digital objects in a virtual gamespace to removeobjects from the virtual gamespace;

FIG. 4C is another exemplary representation of the virtual suction toolinteracting with digital objects in a virtual gamespace to depositpreviously removed objects into the virtual game space to build astructure that aids a movable character in overcoming an obstacle;

FIG. 5A is an exemplary representation of a movable game character in avirtual gamespace encountering an obstacle and a virtual suction toolinteracting with a digital object in a virtual gamespace;

FIG. 5B is another exemplary representation of a virtual suction toolinteracting with digital objects in a virtual gamespace to transform aremoved object into a different form before expelling it back into thevirtual gamespace;

FIG. 5C is an exemplary representation of the expelled transformedobject interacting with the virtual gamespace;

FIG. 6A is an exemplary representation of a plurality of movable gamecharacters in a virtual gamespace encountering an obstacle;

FIG. 6B is another exemplary representation of a virtual suction toolinteracting with digital objects in a virtual gamespace to remove thegame characters;

FIG. 6C is an exemplary representation of a virtual suction toolinteracting with digital objects in a virtual gamespace in aiding aplurality of movable characters in overcoming an obstacle by depositingthem back into the virtual gamespace at different positions, therebyproviding a virtual transporter mechanism;

FIG. 7A is another exemplary representation of a plurality of movablegame characters in a virtual gamespace encountering an obstacle;

FIG. 7B is another exemplary representation of a virtual suction toolinteracting with digital objects in a virtual gamespace to removeobstacles thereby allowing game characters to proceed;

FIG. 7C is an exemplary representation showing how game characters canachieve their goal;

FIG. 8A is an exemplary representation of an enemy character havingcaptured a plurality of movable characters in a virtual gamespace and avirtual suction tool interacting with a digital object;

FIG. 8B is an exemplary representation of a virtual suction toolinteracting with digital objects in a virtual gamespace in aiding aplurality of movable characters to escape capture;

FIG. 9 is an exemplary representation of a plurality of movablecharacters appearing in a virtual gamespace and a virtual suction toolinteracting with digital objects to aid the characters in reaching agoal;

FIG. 10A is an exemplary representation of a virtual suction tool havinga containment indicator;

FIGS. 10B, 10C and 10D are exemplary illustrative non-limitingimplementations of the virtual suction tools with exemplary non-limitingcontainment indicators interacting with digital objects in a virtualgamespace; and

FIGS. 11A and 11B are an exemplary representation of another exemplaryillustrative non-limiting virtual suction tool arrangement interactingwith digital objects in a virtual gamespace.

DETAILED DESCRIPTION

Exemplary Virtual Suction Tool

In one exemplary non-limiting illustrative implementation shown in FIGS.1A-1C, a real world tool or object such as a stylus 101 is used tocontrol game play on a gaming or computing device touch screen S. Afunction of stylus 101 in this particular example is to indicatepositions on a touch screen S. However, in the exemplary illustrativenon-limiting implementation, the stylus 101 appears to the user to havethe property of being able to apply suction to the objects in thevirtual 3D world depicted on screen S.

For example, as shown in FIG. 1 a, the user uses stylus 101 to select anobject 105 by pointing to it. When the user selects such an object,animation on screen S is used to make it appear that stylus 101 ispulling the selected object out of the virtual world across the screenboundary into the real world. In this example, the stylus 101 functionsas a virtual suction device with storage. Unlike a real suction device,however, the exemplary illustrative non-limiting simulated virtualsuction device may have properties which are impractical or impossibleto duplicate in the real world. For example, the simulated device cansuck up and hold solid objects within it, whereas in the real worldobjects that do not conform to the suction device tip might simply stickto a suction hole at the end of the tip. Because the player isinteracting with the virtual world, the game designer can enhance theplayer's virtual tool with an array of capabilities that the playercould not necessarily access in real life.

Once the player has selected the object 105 for manipulation (in thiscase by touching it with the end of the stylus), the stylus 101 beginsto “suck” the object out of the virtual world and into the stylus 101.This realistic nature of this effect is heightened as the object 105 isanimated to appear to begin to distort as it is “drawn into” the stylusas shown in FIG. 1 b. The distortion could be slight visuallyperceptible perturbations, or it could be more radical distortion of theobject as shown in FIG. 1 b. The object 105 distorts in a way that onecould imagine a real object would distort if pulled into a powerfulsuction hose. Additionally, to add to the effect, the game device mayoutput a sound 106. This can be a hissing, suction-like sound 106 in thecase of a vacuum, or can be altered to any appropriate sound.

As the block 105 is fully “drawn into” the stylus 101, it furtherdistorts, as shown in FIG. 1C. Eventually, the block 105 disappears“into” the stylus 101, and the player will have the impression that theblock 105 has left the virtual game world and is now trapped within thestylus 101 in the real world. The player will have the impression thatthe block 105 is now within the stylus 101 the player is holding in hisor her hand. This final suction can be accompanied by a different noise108, such as a popping sound as the object 105 “clears” the virtualsuction tip. The game designer can tailor the noises according to thetool type. The closer the noises are to those that a similar “real” toolwould make, the more the player feels like he is directly interactingwith virtual objects using a real world suction tool.

Exemplary Video Game Platform

In FIG. 2A, an exemplary illustrative non-limiting game apparatus 1 thatcan be used for implementing the virtual suction tool techniquedescribed above. In one exemplary illustrative non-limitingimplementation, system 10 may comprise a Nintendo DS portable handheldvideogame system including a 3D graphics generator capable of generatingcomplex texture-mapped displays of characters interacting with a 3Dworld from any desired viewpoint.

In the exemplary non-limiting illustrative implementation shown, gamedevice 1 includes two liquid crystal displays (LCDs) 11 and 12, whichare accommodated in a housing 18 so as to be located at predeterminedpositions. Specifically, in the case where the first liquid crystaldisplay (hereinafter, referred to as the “LCD”) 11 and the second LCD 12are accommodated in a vertically stacking manner, the housing 18includes a lower housing 18 a and an upper housing 18 b. The upperhousing 18 b is pivotably supported by a part of an upper surface of thelower housing 18 a. The upper housing 18 b has a planar shape slightlylarger than a planar shape of the first LCD 11, and has an opening forexposing a display screen of the first LCD 11 on one main surfacethereof. The lower housing 18 a has a planar shape longer in thehorizontal direction than the planar shape of the upper housing 18 b,and has an opening for exposing a display screen of the second LCD 12 atapproximately the center of the lower housing 18 b in the horizontaldirection. One of two side sections of the lower housing 18 ainterposing the second LCD 12 has speaker holes of a speaker 15, andeach of the two side sections has an operation switch section 14.

The operation switch section 14 includes an operation switch (button A)14 a and an operation switch (button) 14 b which are attached to onemain surface of the side section of the lower housing 18 a which is tothe right of the second LCD 12 as seen in FIG. 2A. The operation switchsection 14 also includes a direction indication switch (cross key) 14 c,a start switch 14 d, and a select switch 14 e which are attached to onemain surface of the side section of the lower housing 18 a to the leftof the second LCD 12 as seen in FIG. 2A. The lower housing 18 a furtherincludes side surface switches 14 f and 14 g, which are respectivelyprovided on the upper surfaces of the side sections of the lower housing18 a to the left and to the right of the second LCD 12. When necessary,further operation switches may be provided, or unnecessary operationswitches may be removed.

On an upper surface (the surface entirely shown in FIG. 2A) of thesecond LCD 12, a touch panel 13 (surrounded by the dashed line in FIG.2A) is provided. The touch panel 13 is of, for example, any of aresistance film system, an optical (infrared) system, and a staticcapacitance coupling system. When a stylus 16 (or a finger) presses,moves on, or touches an upper surface of the touch panel 13, thecoordinate position of the stylus 16 is detected and the coordinate datais output.

In the vicinity of a side surface of the upper housing 18 b, anaccommodation hole (an area represented by the two-dot chain line inFIG. 2A) is formed when necessary for accommodating the stylus 16 foroperating the touch panel 13. In a part of one surface of the lowerhousing 18 a, a cartridge insertion section (an area represented by theone-dot chain line in FIG. 2A) is formed, for detachably accepting agame cartridge 17 (hereinafter, referred to simply as the “cartridge17”) having a built-in memory (e.g., a ROM) which stores a game program.The cartridge 17 is a memory medium storing a game program, and is, forexample, a nonvolatile semiconductor memory such as a ROM or a flashmemory. A part of the lower housing 18 b inner to the cartridgeinsertion section has a built-in connecter (see FIG. 2B) to beelectrically connected with the cartridge 17. The lower housing 18 a (orthe upper housing 18 b) accommodates an electronic circuit board havingvarious electronic components such as a CPU and the like mountedthereon. The memory medium for storing the game program is not limitedto the nonvolatile semiconductor memory, but may be a CD-ROM, a DVD, ora similar optical disc-shaped memory medium.

Next, with reference to FIG. 2B, an internal structure of the gameapparatus 1 will be described. FIG. 2B is a block diagram showing theinternal structure of the game apparatus 1.

In FIG. 2B, an electronic circuit board accommodated in the housing 18has a CPU core 21 mounted thereon. The CPU core 21 is connected to theconnector 28, to be connected to the cartridge 17, via a predeterminedbus, and the CPU core 21 is also connected to an input/output interface(I/F) circuit 27, a first graphic processing unit (first GPU) 24, asecond graphic processing unit (second GPU) 26, and a working RAM (WRAM)22.

To the connecter 28, the cartridge 17 is detachably connectable. Asdescribed above, the cartridge 17 is a memory medium for storing a gameprogram. Specifically, the cartridge 17 has a ROM 171 storing the gameprogram and a RAM 172 rewritably storing backup data mounted thereon.The game program stored in the ROM 171 in the cartridge 17 is loaded onthe WRAM 22, and the game program loaded on the WRAM 22 is executed bythe CPU core 21. Temporary data and data for generating an image whichare obtained by the CPU core 21 through execution of the game programare stored in the WRAM 22.

As described above, the ROM 171 stores a game program, which is a groupof instructions and a group of data in the format executable by thecomputer of the game apparatus 1, especially by the CPU core 21. Thegame program is read into and executed by the WRAM 22 when necessary. Inthis embodiment, the game program and the like are recorded in thecartridge 17, but the game program and the like may be supplied byanother medium or via a communication network.

The I/F circuit 27 is connected to the touch panel 13, the operationswitch section 14, and the speaker 15. The speaker 15 is located at aposition just inside the speaker holes described above.

The first GPU 24 is connected to a first video RAM (hereinafter,referred to the “VRAM”) 23, and the second GPU 26 is connected to asecond VRAM 25. In accordance with an instruction from the CPU core 21,the first GPU 24 generates a first game image based on data forgenerating an image stored in the WRAM 22 and draws the first game imagein the first VRAM 23. In accordance with an instruction from the CPUcore 21, the second GPU 26 generates a second game image based on datafor generating an image stored in the WRAM 22 and draws the second gameimage in the second VRAM 25.

The first GPU 24 is connected to the first LCD 11, and the second GPU 26is connected to the second LCD 12. In accordance with an instructionfrom the CPU core 21, the first GPU 24 outputs the first game imagedrawn in the first VRAM 23 to the first LCD 11. The first LCD 11displays the first game image which is output from the first GPU 24. Inaccordance with an instruction from the CPU core 21, the second GPU 26outputs the second game image drawn in the second VRAM 25 to the secondLCD 12. The second LCD 12 displays the second game image which is outputfrom the second GPU 26.

The I/F circuit is a circuit for exchanging data between externalinput/output devices such as the touch panel 13, the operation switchsection 14, the speaker 15 and the like, and the CPU core 21. The touchpanel 13 (including a device driver for the touch panel 13) has acoordinate system corresponding to a coordinate system of the secondVRAM 25, and outputs coordinate position data corresponding to theposition which is input (indicated) by the stick 16 or the like. Theresolution of the display screen of the second LCD 12 is, for example,256 dots×192 dots, and the detection precision of the touch panel 13 is256 dots×192 dots in correspondence with the resolution of the displayscreen of the second LCD 12. The precision detection of the touch panel13 may be lower or higher than the resolution of the display screen ofthe second LCD 12.

FIG. 2C is a flowchart of an exemplary illustrative non-limitingimplementation of software-controlled virtual tool functionality. In theexample shown, software can be executed to control system 10 to performa video game or other exemplary application. As described above,software may reside on a removable memory card 28 or other storagedevice insertable into system 10, or it may be provided to the systemvia other means (e.g., preloading into internal flash or othernon-volatile memory, transmission over a wireless connection via WiFifunctionality 33, or any other convenient means).

Referring to the exemplary FIG. 2C flowchart, upon starting a new gameor other application, the software controls system 10 to initialize thegame and display 12, 22 (block 502) and to then display a 3D worldincluding objects (block 504). The software then controls system 10 todetect the position of stylus 24 on touch screen display 22 as well asthe state of the various other control inputs 20 (block 506).

If, based upon the current game state, the other control inputs and theposition of the stylus 24, the software determines that the stylus is toact as a virtual suction tool (decision block 508), then an exemplaryillustrative non-limiting implementation, the software further controlssystem 10 to animate the character or other object directly beneath theposition of stylus 24 to make it appear that the object is being drawninto the virtual suction tool stylus 24 (block 512). The object may thusdeform in a manner that is based upon the relative position of thestylus 24 tip with respect to the object. Different objects may deformdifferently depending on the type of matter they represent (e.g., moresolid objects may not deform at all but may simply be sucked into thevirtual sucking tool stylus 24 in a non-deformed appearance, whereasother, more flexible objects may be deformed extensively under theeffect of the virtual suction before disappearing from the screendisplay 22) (block 512).

At the same time, the software controls system 10 to generate awhooshing or suction sound that emanates from speaker 32 to create theillusion of virtual suction. When the object has animated under theillusion of suction force emanating from virtual suction tool stylus 24finally disappears “into” the stylus, a “pop” or other suitable soundmay emanate from speaker 32 to create the further illusion that theobject has been sucked into the virtual suction tool stylus 24 (block514). Game play may then proceed by other characters and/or objects(block 510). A similar action of blocks 508, 512, 514 may be used tocontrol the virtual suction tool stylus 24 to apply reverse suction toexpel objects into the virtual 3D world displayed on displays 12, 22.

The following describes particular non-limiting illustrative examples ofhow the virtual suction tool mechanism described above can be used invarious game play scenarios.

Example use of Virtual Suction Device to Move Objects From One Positionto Another—Virtual “Transporter” Tool

In the exemplary non-limiting illustrative implementation shown in FIGS.3A-3C, a game character 100 moving in a direction shown by arrows 103encounters an obstacle 107 (in this case a chasm). The character 100cannot cross the obstacle 107 if the gamespace is left as it isdisplayed in FIG. 3A. The character 100 has, however, just crossed overa series of movable digital objects, in this case blocks 105.

As shown in FIG. 3B, the player can use the virtual suction toolimplemented by the stylus 101 to capture the blocks. Although thecapture is virtual, the stylus 101 may appear to suck the blocks 105 offof the screen, and may be accompanied by a suitable sound effect asexplained above. After the blocks 105 are no longer displayed on thescreen, the player feels they are “stored” within the stylus 101 held bythe player (in fact, the exemplary illustrative stylus 101 cannot storeanything, and the virtual storage is being performed by a memory devicethat “remembers” what the virtual vacuum tool has suctioned up). Movingthe stylus 101 can be used to move the “stored” blocks 105 around in thereal world, so that they can be re-deployed in a new location in thevirtual game world.

As shown in FIG. 3C, the player uses the stylus 101 to re-deploy theblocks 105 to build a bridge such that the obstacle 107 can be crossedby the character 100. Again, the stylus 101 may appear to inject theblocks 105 from the tip of the stylus 101, providing the illusion thatthe stylus 101 had actually been holding the blocks 105. The virtualsuction tool may be placed into a “reverse suction” mode to expel the“retained” objects at desired positions with the virtual world. Asuitable “exhaust” type sound effect may be played to aid in thisillusion. The game character may now cross the bridge and proceed.

A game may determine whether a stylus should pick up or deploy a blockin a variety of manners. In one exemplary non-limiting illustrativeimplementation, the stylus will pick up an object if it does notcurrently have any (or only a certain number of) objects in storage, andwill deploy any objects it is currently storing. In this exemplaryimplementation, the stylus is made to resemble a suction tool with afinite capacity, such as a vacuum cleaner. In another exemplarynon-limiting illustrative implementation the stylus will be operable topick up or deploy objects at any time, and may use a FIFO or LIFO queueor a random or pseudo random order to redeploy objects. According tothis exemplary implementation, the stylus has capabilities not found inany real world tool, namely the capacity for nearly infinite storage ofpotentially huge objects. In this exemplary implementation, if thestylus is touched to or encounters a point on the screen displaying amovable object, the stylus will capture that objects, and if the stylusis touched to or encounters a point on the screen where an object can bere-deployed, it will re-deploy the object in accordance with itsdeployment algorithm.

Additional controls 20 may be used to control the suction/reversesuction mode of the virtual suction tool, or on-screen controls can bedetected, or strokes or gestures can be recognized to control virtualsuction tool mode of operation. The stylus may also contain varioustypes of objects and deploy appropriate types in appropriate spaces onthe screen, such as only deploying characters if the deployment isselected over ground or only deploying blocks if the deployment isselected over a pit of spikes. The game space can be designed so thatsome objects are subject to suction effects and others are fixed andimpervious to suction. Any suitable method for determining styluscapture/deployment strategy may be implemented.

Exemplary use of Virtual Suction Tool to Consutruct Structures Withinthe Virtual Enviroment

In another exemplary non-limiting illustrative implementation shown inFIGS. 4A-4C, a character 100 encounters an obstacle 109 that is too tallfor the character 100 to jump over or onto. Again, the character 100 hasjust crossed over a set of movable digital objects 105 in the directionindicated by the arrows 103.

As shown in FIG. 4B, the player can use the virtual suction tool ofstylus 101 to capture the blocks 105. Once the blocks 105 are captured,the player can, as shown in FIG. 3C, re-deploy the blocks 105 in a steppattern using the stylus 101, and the character 100 can climb thesesteps to overcome the obstacle 109.

Exemplary use of Virtual Suction Tool to Transform or Transmute Objects

FIG. 5A shows another non-limiting exemplary implementation in which acharacter 100 encounters a large crate-like object 113. There are noblocks available for stacking, but there is a single red block 111(color not shown) on the screen which the player can capture with thestylus 101. This block 111 may be displayed in a different color thanthe stackable blocks 105 shown in FIGS. 2 and 3 to indicate itsdifferent properties.

After capturing the block 111, the player re-deploys the block 111 ontothe obstacle 113 using the stylus. As shown in FIG. 5B, the virtualsuction tool redeploys the block 111 it as a fire ball 115 instead of asa block. The flame 115 then burns away the obstacle 113 as shown in FIG.5C. The virtual suction tool thus has the property in this example oftransmuting a block into a fireball. Alternatively, the virtual suctiontool of stylus 101 can be thought of as turning into a match, torch ofother fire lighting tool after it has sucked up or ingested a red block111.

Example use of Virtual Suction Tool to Transport or Rescue Characters

Many of us have watched Star Trek scenes where a character in troublecan radio to be “beamed up” and transported away from trouble. A similareffect can be provided using the virtual suction device stylus 101. In afurther exemplary non-limiting illustrative implementation shown in FIG.6A, a plurality of characters 100 encounters an obstacle 107. There areno blocks for the player to manipulate in this instance so the situationappears hopeless.

Instead of manipulating blocks, the characters 100 become movabledigital objects and the player captures them with the stylus 101 asshown in FIG. 6B. FIG. 6C shows the player using the stylus 101 tore-deploy the characters 100 safely on the other side of the obstacle107. Now the player has actually drawn the game characters out of thegame and “brought” them into the real world temporarily. The stylus 101acts somewhat like a virtual transporter tool in Star Trek. The playercan replace the game characters back into the virtual world when a safearea is found.

Exemplary use of Virtual Suction Tool to Remove Objects at AppropriateTimings

In an additional exemplary non-limiting illustrative implementationshown in FIG. 7A, a plurality of characters 100 are prevented fromreaching a goal 117 by a series of movable digital objects 105.

In FIG. 7B, the virtual suction tool stylus 101 is shown capturing thedigital objects 105, allowing the characters 100 to fall through theresulting gap at just the right time. The characters 100 can now allreach the goal 117. The virtual suction tool stylus 101 has thuspermanently removed objects from the virtual gamespace. In this case,the stylus 101 is used to activate a first character in FIG. 7A. Thefirst character activates the second character, and the virtual suctiontool stylus 101 sucks up blocks when the first character arrives rightabove the third character. The third character is activatedautomatically, and all three characters proceed to the goal (chainreaction).

Exemplary use of Virtual Tool to Suction Objects and Transform them intoAdditional Animated Tools for Injection into the Virtual Enviroment

FIG. 8A shows a further exemplary non-limiting illustrativeimplementation in which an enemy 121 has captured a plurality ofcharacters 100 with fishing lines 123. A series of movable digitalobjects 119 are also shown. The objects may be of a certain color orappearance to indicate a particular property. The player can capturethese objects using the stylus 101 operating as a virtual suction tool.

FIG. 8B shows the player re-deploying a captured digital object 119using the stylus 101 virtual suction tool in the exhaust mode. Theobject 119 is transformed during deployment into a moving cutter 125that cuts the line 123. One of the plurality of characters 100 is nowfree from the line. During this operation, the player has to think aboutthe position and timing of the cutter 125. The game character isdestroyed if he is dropped on a hazard or falls from a very highposition. The player also pays attention to which block is taken. Forexample, if all of the blocks are sucked up into the virtual suctiontool, there will be no platform for the game character to land on.

Example use of Virtual Suction Tool to Move Objects to Catch OtherObjects

FIG. 9 shows another exemplary non-limiting illustrative implementation.In this implementation, a plurality of characters 200 are falling fromholes 201. The characters 200 are moving towards a goal 207, but longfalls through gaps such as a gap 209 will cause the character 200 todie. The player can use the virtual suction tool stylus 101 to draw themovable objects 205 to different parts of the virtual world to ensurethat the characters 200 all make it to the goal 207 without falling toofar. In the top of FIG. 9, a game character drops from the upper screenone after another. The virtual suction tool stylus 101 is used to moveblocks to catch the game character. If timing of movement is not right,the game character may drop into a pot hole and break.

Virtual Suction Tool with Visual Indicator

FIG. 10A shows an exemplary non-limiting illustrative implementation ofa stylus 301 adapted for digital object capture. The stylus 301 may beprovided with a button 311 and a digital display 313 or some othersuitable method for representing stylus contents and capacity. Thevirtual suction tool stylus 301 sucks up blocks and the player can seevisually the number and color of the blocks “inside” the pen.

The stylus 301 may be attached to a device hook-up 305 by a cable 303.Alternatively, the stylus 301 may connect wirelessly to the devicehook-up 305, and could be powered by internal batteries, solar cells,etc. The display or indicator incorporated into stylus 301 may be anarray of light emitting diodes, a linear or elongated color ormonochrome liquid crystal display, a thermo-sensitive display, or anyother type of display. The display can indicate number of “contained”objects and/or characteristics of the contained objects (e.g., color,size, etc.).

FIG. 10B is an exemplary non-limiting illustrative implementation of theadapted stylus 301 capturing digital objects 105. The objects aredisplayed as virtually captured 307 in the stylus 301 by the display313. A thermometer type virtual level indicator is employed to indicateto the user how “full” the virtual suction tool is. Virtual suction maycease if the virtual suction tool indicator indicates the tool iscompletely “full.”

FIG. 10C shows another exemplary non-limiting illustrativeimplementation of a further adapted stylus 350. This stylus 350 has beenadapted to work with a particular game and actually show digital images309 of various digital objects 100 that have been captured. An LCD ordisplay or other suitable methods of displaying contents and capacity ofthe stylus may be used to display images of the captured objects so theplayer can see which objects he has captured and which objects will beexpelled next and in which order. This provides the player with a bettervisual inventory of the objects contained within the stylus 350, andcreates a highly visual sense that the virtual suction tool has indeedcaptured the suctioned objects. Each time the virtual tool sucks up anobject, it is displayed at the bottom of the display closet to the tipand all the other objects move up in the display to make it appear thatthey have been displayed. The resulting display thus appears to theplayer to be a window into an object reservoir within the virtualsuction tool—much as an eye dropper allows you to see how much liquidyou have suctioned into the eye dropper's tube. Any number of suitableadaptations may be made to this stylus to make it more generic indisplay or more game specific.

FIG. 10D shows a further exemplary non-limiting illustrativeimplementation of the stylus 350 re-deploying captured objects 100. Thevolume of captured objects displayed 309 decreases as the objects 100are released to the screen. The button 311 may be depressed to indicateddeployment rather than capture. The button 311 can be depressed to expelan item or character “within” the stylus 301. If the player wants toreplace items in the pen rapidly, he/she can hold the button and sweepthe screen so that multiple items will be deployed or “sprayed” onto thescreen. If the player likes precise control, he/she can stamp items oneby one by tapping the screen.

Exemplary Non-Touch Based Corral Type Pointing Device Virtual Suction orSweeping Tool

FIG. 11A shows an exemplary non-limiting illustrative implementation ofa game controller 323 with object capture capability. This controller323 includes a pointing capability as described in U.S. patentapplication Ser. No. 60/716,937 (723-1706) incorporated herein byreference. The controller 323 may be used as a virtual suction tool, avirtual sweeping tool, or other virtual tool with object capturecapability. Controller 323 may capture objects from a display screen 325out of reach of the player, and may show an inventory of capturedobjects on a mini-display 321. The objects 121, 200 are displayed on themain screen 325 and a target reticule 327 may be employed to aid indetermining which object is about to be captured. The objects can beanimated to move as they are displayed on the mini-display 321 to makeit appear as if they are being held in a handheld corral.

FIG. 11B shows a further exemplary non-limiting illustrativeimplementation of a game controller 323 having captured a plurality ofvirtual objects 100, 200, 121, 400, 500. In this implementation, theplayer has captured several characters 100, 200, an enemy 121, and someitems 400, 500. All of the captured items 100, 200, 121, 400, 500 aredisplayed on a screen 321. Any suitable display may be used to indicatedstored objects.

FIG. 11B shows that the player may store and organize characters and/oritems in the game without opening a separate window. Players don't haveto miss the contents of the current game screen. This can be used as aconvenient “transportation” or “transporter” tool for games whichrequire managing a lot of characters and/or items.

In a further exemplary non-limiting illustrative implementation,controls may be provided on the game controller 323 and the player canplay a mini-game using the captured objects. For example, the playercould manipulate a character 100 to grab object 400 and attack enemy121. The player could then have to manipulate all characters 100, 200out through object (pipe) 500 to escape. Any number of suitablescenarios can be provided for the stored objects.

We have described technology that provides interesting user interfacesto increase the fun and excitement of videogame play. Using a pointingdevice to interact with a 2D or 3D graphics-based system, it is possibleto simulate the behavior of a virtual directional tool capable of removematerial from and/or injecting material into the 3D or other graphicsenvironment. The use of virtual suction or vacuum provides a veryintuitive virtual suction tool capability that videogame users readilyunderstand. A simple pen-based stylus or other pointing device held inthe user's hand thus appears to have the capabilities of a virtualvacuum cleaner or other virtual suction device that can remove objects,game characters, water, fog, fire or any other material from the virtualenvironment. The user has the feel that the virtual suction device hasabsorbed and now contains or has otherwise disposed of the material thathas been “sucked” from the virtual environment into a real world objectheld in the player's hand.

In some implementations, the game or application developer may wish tocreate the impression that the material that has been virtually suckedfrom the virtual game environment has been disposed of (i.e., via aninvisible virtual vacuum hose connecting the stylus to a wastereceptacle). In other exemplary illustrative non-limitingimplementations, the virtual suction tool may be designed by the game orother application developer to virtually contain the objects that havebeen sucked from the virtual environment so they be expelled back intothe environment. For example, in the order in which there were initiallyremoved. In some exemplary illustrative non-limiting implementations,the objects sucked from the virtual environment may be returned to theenvironment in the same form in which there were removed. In otherexemplary illustrative non-limiting implementations, the virtual suctiontool may transform the objects (e.g., a block may be transformed intofire, water may be transformed into ice cubes, etc.). before beingreinserted into the virtual environment.

While suction is a useful and interesting effect to provide to a virtualtool, other ways the virtual tool may interact with the environment mayalso be provided alternatively or in addition. As one example, a virtualtool may be provided with the functionality of a virtual air compressorso that it can exhaust material into the virtual game environment.Virtual power washers, virtual flame throwers, virtual brooms or othersweeping devices, virtual eye droppers, virtual laser beams or otherweapons, virtual cutting tools, or any other number of arrangements arepossible.

While it is useful or advantageous to provide direct contact between thetip of a virtual tool stylus and a touch screen displaying the virtualenvironment the virtual tool is interacting with, other arrangements arepossible. For example, as has been explained above, it is possible touse a remote pointing device aimed at a display to provide virtual toolinteraction. Such virtual tool interaction may be provided with adisplay capability so the user may see, in a handheld device display,images of the objects and other material he or she has captured from thevirtual environment.

While the technology herein has been described in connection withexemplary illustrative non-limiting implementations, the invention isnot to be limited by the disclosure. For example, although specificexemplary non-limiting implementations herein relate to using a stylusto simulate a virtual suction tool interacting with 3D touch-screenbased video games, other implementations are possible. For example,applications outside of the gaming arena, including simulations, virtualreality and augmented reality, are possible. While 3D graphics providesadvantages, other forms of graphics (e.g., 2D, holographic, television,digital television, etc.) could be used instead. Vacuum or suction toolsprovide interesting results, but in other implementations the virtualtool could perform the function or effect of any tool within the realworld or within the imagination of the human mind. While the use of atouch screen based interface has advantages, other implementations coulduse non-touch screen based approaches to indicate position,interactivity, or the like. While the exemplary illustrativenon-limiting implementation uses the same physical object to bothinteract with the computer controlling or creating the virtual 3Denvironment and to be provided with simulated and/or enhanced toolfunctionality, other approaches are possible. While the use of a stylusis useful in certain applications, other forms of physical objects(e.g., objects made to resemble the tool they are simulating) could beused instead. The actual function of the object in the real world canhave a range from virtually no functionality to extensive functionality.For example, in some exemplary non-limiting implementations, the objectused for interactivity could be an actual tool having extensive toolfunctionality, with the virtual environment providing a degree ofenhancement to an existing real world tool function and/or adding a new,virtual function to a repertoire of actual real world functions. Theexemplary illustrative non-limiting implementation animates certainobjects being deformed as they are suctioned into the virtual vacuumtool, but in other exemplary implementations no deformation is shown andthe objects are suctioned “as is” into the tool, or certain objects(e.g., those with flexible virtual characteristics) may be deformedwhereas other objects (e.g., “hard” virtual objects such as stones, gamecharacters, etc.) may change appearance in ways other than deformation(e.g., reduce in size before being suctioned so as to appear to “fit”dimensionally into the virtual suction tool tip) or not changeappearance at all. The invention is intended to be defined by the claimsand to cover all corresponding and equivalent arrangements whether ornot specifically disclosed herein.

1. In a puzzle game played using a touchscreen display, said puzzle gameof the type that displays a virtual world including at least oneobstacle on said touchscreen display and allows a human player to use astylus to interact with the puzzle game, the stylus having a tip, amethod for permitting the human player to manipulate at least onevirtual object on the touchscreen display to help a game characternavigate through said virtual world and overcome the at least oneobstacle, the method comprising: displaying at least one virtual objecton the touchscreen display; allowing the human player to use the stylusto interact with the display and puzzle game including simulating asuction and expulsion function of the stylus at least in part via thedisplay; deforming the shape of the at least one displayed virtualobject based at least in part on the simulated suction functionality ofthe stylus so that as the human user moves the tip of the stylus overthe at least one virtual object, the virtual object disappears in a waythat creates an impression that the stylus is sucking the virtual objectup into an internal cavity or reservoir within the stylus by deformingportions of the displayed virtual object closest to the stylus so thatthe object disappears from the display; and at least in part in responseto the human user moving the stylus tip on the touchscreen, providing afurther display on the touchscreen that makes it appear that the stylusis injecting the virtual object from the tip of the stylus into thevirtual world, the injected virtual object providing a structure withinthe virtual world that allows the game character to overcome the atleast one obstacle.
 2. The method of claim 1 wherein the simulatedfunctionality of the stylus provides capabilities beyond that of a realworld equivalent of a stylus.
 3. The method of claim 1 wherein thesimulated functionality of the stylus does not have a real worldequivalent.
 4. The method of claim 1 wherein the stylus tip interactswith the puzzle game by touching the portion of the touch screen thatdisplays the virtual object.
 5. A method of controlling a video gamehaving a video game character that uses a virtual structure within thevideo game to overcome challenges, wherein a user holds a real-worldtool for manipulating the displayed virtual structure, said methodcomprising the steps of: displaying a the virtual structure on atouchscreen; processing image data representing said virtual structureto simulate a 3D operation on said virtual structure by said real-worldtool held by said user; and modifying the displayed three-dimensionalappearance of said virtual structure to simulate said 3D operation ofsaid tool on said virtual structure, thereby facilitating use of thevirtual structure by the video game character to overcome the challenge.6. A video game system of the type that presents a video game characterwith a challenge, comprising: a 3D graphics generator that generates 3Dgraphics for display; a touch screen display device coupled to saidgraphics generator, said touch screen display device displaying said 3Dgraphics and sensing touch; and a software-controlled processor coupledto said touch screen display and to said graphics generator, saidsoftware-controlled processor simulating suction applied from the realworld to 3D graphics displayed on said touch screen display to remove atleast one displayed 3D object based at least in part on said sensedtouch and to inject said removed at least one 3D object in anotherposition within said game to thereby provide a structure that the videogame character within the game uses to overcome the challenge.
 7. Thesystem of claim 6 further including a stylus for interacting with saidtouch screen display, said stylus in conjunction with functionsperformed by said software-controlled processor comprising a virtual 3DSuction tool.
 8. The system of claim 6 wherein said software-controlledprocessor further simulates virtual 3D reverse suction to allow materialvirtually existing in the real world to be expelled into a 3D scenedepicted on said touch screen display.